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EXECUTIVE  SUMMARY 


OBJECTIVES  OF  THE  DEMONSTRATION 

This  project  was  undertaken  by  White  River  Technologies,  Inc.  (WRT)  to  assess  the  feasibility  of 
acquiring  classification  quality  data  during  dynamic  Electromagnetic  Induction  (EMI)  surveys  of 
Munitions  Response  Sites  (MRS).  A  dynamic  classification  survey  was  performed  using  the  One 
Pass  Time  domain  EM  Array  (OPTEMA)  at  the  former  Southwestern  Proving  Ground  near  Hope, 
AR.  Over  the  course  of  6  days  in  September,  2015  the  team  surveyed  4  acres  of  Recovery  Eield 
(RE)  15.  Survey  activities  included  mobilization  of  equipment  to  the  site,  initial  calibration  and 
instrument  verification  activities,  dynamic  data  collection  over  the  4-acre  area,  daily  instrument 
verification  and  data  quality  checks,  and  demobilization. 

TECHNOLOGY  DESCRIPTION 

The  OPTEMA  sensor  comprises  an  array  of  five  multi-directional  transmitters  and  14  receivers 
that  are  optimally  configured  to  provide  EMI  characterization  across  the  entire  1.8  meter  sensor 
swath.  This  capability  is  the  basis  for  effective  dynamic  classification  since  sensor  position  during 
dynamic  surveys  is  based  on  survey  transects  rather  than  on  a  priori  target  location.  In  contrast  to 
the  sensor  positioning  requirements  for  cued  surveys,  it  is  likely  that  a  large  number  of  targets  will 
be  located  at  some  lateral  offset  relative  to  the  array  center  during  a  dynamic  survey.  Therefore, 
high  quality  EMI  characterization  across  the  array  swath  is  critical  for  successful  classification. 

DEMONSTRATION  RESULTS 

During  the  4-acre  survey,  a  production  rate  of  approximately  0.5  acre/hour  (hr)  was  achieved. 
Instrument  verification  was  performed  three  times  daily  in  the  Instrument  Verification  Strip  (IVS) 
to  confirm  sensor  functionality.  Quality  checks  included  evaluation  of  system  positioning  accuracy 
and  data  sample  density,  as  well  as  evaluation  of  classification  feature  quality  (obtained  from  IVS 
data  analysis).  These  quality  checks  ensured  that  the  team  departed  the  site  with  classification- 
level  data  that  met  the  following  objectives: 

1 .  100%  coverage  of  the  4-acre  site  with  1 .2m  transect  spacing  (33%  sensor  footprint  overlap) 
and  average  along  track  sample  spacing  <8  cm; 

2.  Dynamic  sensor  noise  levels  low  enough  to  enable  classification  of  munitions  as  small  as 
20mm  projectiles; 

3.  Positioning  accuracy  sufficient  to  determine  target  source  locations  to  within  15  cm  of 
actual  ground  truth  locations. 

Post-survey  data  analysis  activities  included  dipole  model-based  inversion  of  the  dynamic  data  to 
acquire  classification  features  associated  with  anomalies.  Analyzing  these  features,  classified  2022 
anomalies  that  were  intrusively  investigated  by  dig  teams  for  ground  truth  verification.  Of  the  2022 
anomalies,  29  were  Targets  of  Interest  (TOI)  and  the  remaining  items  were  non-hazardous  debris. 
Independent  scoring  of  the  classification  analysis  revealed  that  it  correctly  classified  100%  of  the 
TOI  with  a  clutter  rejection  rate  of  94%  and  it  achieved  a  clutter  rejection  rate  of  82%  at  the  stop 
dig  point.  These  results  indicate  that  dynamic  classification  methods  provide  an  efficient  data 
collection  alternative  to  cued  surveys  and  can  produce  classification  results  comparable  in  quality 
to  those  obtained  with  cued  methods. 
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IMPLEMENTATION  ISSUES 


Over  the  course  of  the  survey,  a  few  data  collection  issues  were  encountered  that  required 
corrective  actions.  These  issues  included  loose  receivers  that  created  elevated  noise  in  some  of  the 
late  time  channels,  GPS  positioning  errors  caused  by  a  cow  knocking  over  the  base  station,  and 
the  failure  of  one  of  the  tow  sled  axles.  These  issues  were  quickly  identified  using  in-field  data 
quality  control  (QC)  checks.  Corrective  actions  included  data  recollects  for  affected  data  files  and 
minor  hardware  repairs.  These  issues  did  not  have  a  significant  impact  on  production  rate. 
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1.0  INTRODUCTION 


This  project  was  undertaken  by  White  River  Technologies,  Inc.  (WRT)  to  assess  the  feasibility  of 
acquiring  classification  quality  data  during  dynamic  Electromagnetic  Induction  (EMI)  surveys  of 
Munitions  Response  Sites  (MRS).  During  the  first  stage  of  this  project,  a  dynamic  classification 
system  known  as  the  One-Pass  Time-Domain  EMI  Array  (OPTEMA)  was  demonstrated  at  the 
Aberdeen  Proving  Ground  (APG)  standardized  Unexploded  Ordnance  (UXO)  Technology 
Demonstration  Site.  Results  from  this  demonstration  indicated  that  it  is  possible  to  achieve  high 
quality  classification  data  from  a  dynamic  survey  system  [1].  To  further  elucidate  the  capabilities 
and  limitations  of  dynamic  classification  methods/systems  applied  to  relevant  “live  site” 
environments,  an  additional  survey  was  performed  using  the  OPTEMA  system  at  the  Eormer 
Southwestern  Proving  Ground  near  Hope,  Arkansas. 

The  process  of  obtaining  both  detection  and  classification  data  during  a  single  dynamic  survey  is 
known  as  the  one-pass  method  of  data  collection.  Currently,  standard  practice  for  classification 
surveys  incorporates  a  two-step  data  collection  process  where  a  preliminary  detection-level  survey 
is  performed  as  part  of  the  Digital  Geophysical  Mapping  (DGM)  operations.  The  DGM  data  serve 
as  a  basis  for  identifying  anomalies  (i.e.,  target  picking)  that  are  subsequently  interrogated  during 
a  secondary  survey  by  static  (or  cued)  EMI  measurements. 

Commercial  sensors  such  as  the  Geometries  MetalMapper  are  optimized  for  cued  surveys  and, 
accordingly,  have  demonstrated  significant  success  when  applied  in  this  two-step  process.  While 
this  cued  mode  is  extremely  effective  for  distinguishing  Munitions  and  Explosives  of  Concern 
(MEG)  items  from  non-hazardous  objects,  it  has  two  significant  drawbacks.  Eirst,  incorporating  a 
secondary  cued  EMI  survey  creates  additional  costs.  While  the  cued  survey  is  more  efficient  than 
excavation  of  all  anomalies,  it  is  typically  more  time  consuming  than  the  preliminary 
detection/DGM  survey  and  thus  adds  significantly  to  the  survey  costs  allocated  for  the  project. 
Second,  because  the  DGM  and  cued  portions  of  the  survey  are  performed  sequentially,  there  can 
be  a  disconnect  between  the  target  picking  process  and  the  cued  survey  process.  Inaccuracies  in 
the  target  picking,  particularly  in  high  anomaly  density  areas,  can  lead  to  sub-optimal  placement 
of  the  sensor  during  the  cued  survey.  Without  accurate  localization  of  the  anomaly  source,  cued 
sensor  classification  performance  can  be  compromised. 

The  one-pass  method  offers  improvements  over  the  cued  method  in  regards  to  these  limitations. 
Detection  and  classification  achieved  from  a  single  EMI  survey  is  efficient  and  could  significantly 
reduce  the  costs  associated  with  the  burden  of  an  additional  cued  survey.  Eurthermore,  using  high 
resolution  classification  sensors  for  detection  and  identification  of  anomalies  could  ultimately 
provide  better  initial  characterization  of  the  target  space  than  that  typically  afforded  by  other  DGM 
sensors.  This  improved  characterization  could  lead  to  better  classification  results  in  high  density 
areas. 

In  conducting  this  project,  the  primary  objective  is  to  determine  whether  it  is  possible,  using  a  one- 
pass  survey  mode,  to  achieve  classification  results  comparable  to  those  achieved  during  cued 
surveys.  Because  of  the  anticipated  lower  operating  costs  associated  with  one-pass  surveys, 
comparable  classification  quality  would  indicate  that  one-pass  survey  modes  may  offer  a  better 
solution  than  cued  surveys  for  many  projects. 
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1.1  BACKGROUND 


The  practical  basis  for  munitions  classification  is  founded  on  the  development  of  advanced  EMI 
sensors  that  provide  high  spatial  and  temporal  resolution  data.  The  data  produced  by  these  sensors 
enable  the  application  of  advanced  physical  models  to  extract  useful  classification  parameters  that 
correspond  to  physical  properties  of  the  object.  The  initial  intent  of  these  classification 
technologies  was  to  provide  a  means  for  inserting  additional  steps  into  the  existing  data  collection 
and  processing  workflows  for  Munitions  Response  (MR)  projects  without  altering  the  established 
protocols.  Accordingly,  the  cued  survey  process  was  developed  as  an  add-on  to  the  existing 
geophysical  survey  workflow  for  such  projects.  By  relying  on  the  standard  DGM  survey  data  for 
target  picking,  advanced  sensors  could  be  incorporated  in  cued  mode  into  the  existing  process  with 
minimal  impact  to  the  overall  flow.  This  process  has  demonstrated  significant  success  for 
discriminating  Targets  of  Interest  (TOI)  from  clutter  at  demonstration  and  production  sites  [2]. 

With  the  gradual  trend  towards  acceptance  of  classification  technologies  in  the  production 
environment,  the  possibility  now  exists  for  shifting  the  focus  of  classification  technology 
development  from  improved  performance  to  improved  efficiency  and  feasibility.  One-pass 
detection  and  classification  may  provide  the  greatest  return  on  investment  for  future  classification 
technology  development.  One-pass  surveys  could  effectively  reduce  excavation  costs  without 
significantly  increasing  geophysical  survey  costs. 

Given  the  recent  successes  of  cued  EMI  sensors  and  the  potential  benefits  of  dynamic 
classification,  several  demonstrations  have  been  performed  to  evaluate  the  effectiveness  of 
applying  cued  EMI  sensors,  such  as  the  Time  domain  EMI  Towed  Array  Detection  System 
(TEMTADS)  2x2  variant  and  MetalMapper  advanced  sensors,  in  a  dynamic  mode.  These 
demonstrations  have  typically  combined  a  dynamic/cued  approach  where  the  dynamic  surveys  are 
used  primarily  to  reduce  the  number  of  cued  reacquisitions  rather  than  to  provide  the  level  of 
classification  necessary  to  eliminate  the  cued  survey  entirely.  While  this  reduction  in  cued 
reacquisitions  represents  some  cost  improvements,  additional  cost  benefits  would  be  realized  with 
complete  elimination  of  any  follow-on  cued  activities  (i.e.,  one-pass). 

The  OPTEMA  was  developed  for  the  specific  purpose  of  acquiring  classification-level  data  in  a 
one-pass  dynamic  survey  mode.  Consequently,  the  OPTEMA  comprises  certain  features  that  are 
designed  to  produce  high  quality  classification  data  during  dynamic  surveys.  Details  of  these 
design  elements  are  presented  in  Section  2. 

1.2  OBJECTIVE  OF  THE  DEMONSTRATION 

The  primary  objective  of  this  demonstration  is  to  quantify  the  detection  and  classification 
performance  of  the  OPTEMA  sensor  operating  in  one-pass  dynamic  mode.  By  demonstrating 
the  system  at  the  former  Southwestern  Proving  Ground  (SWPG),  a  dynamic  data  set  was 
acquired  that  made  it  possible  to  quantify  the  detection  and  classification  performance  of  the 
OPTEMA  with  performance  metrics  (e.g.,  probability  of  detection,  probability  of  false  alarm, 
etc.)  corresponding  to  munitions  types  found  in  a  relevant  environment.  Specifically,  a 
dynamic  survey  was  performed  of  approximately  four  acres  within  the  former  proving 
ground  that  contained  seeded  items,  native  clutter,  and  potentially  native  MEG  as  well. 
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The  intent  of  this  demonstration  was  to  determine  if  the  OPTEMA  could  provide,  in  one-pass 
dynamic  mode,  detection  and  classification  performance  that  would  provide  significant  cost 
savings  relative  to  both  cued  classification  surveys  and  combined  dynamic/cued  classification 
surveys.  If  so,  this  one-pass  technology  would  represent  a  potentially  significant  improvement 
in  operational  efficiency  for  classification  surveys. 

1.3  REGULATORY  DRIVERS 

Demonstration  of  this  technology  elucidates  the  potential  cost  savings  to  munitions  remediation 
projects  as  a  result  of  decreased  time  and  labor  devoted  to  classification  surveys.  DoD  directives 
for  munitions  response  projects  now  include  scope  for  classification  technologies  at  a  number  of 
sites.  It  is  expected  that  over  the  next  5-10  years,  the  list  of  sites  amenable  to  classification  practices 
will  increase  significantly.  Consequently,  any  technologies  that  streamline  operations  associated 
with  munitions  classification  could  have  a  large  impact  on  the  DoD’s  ability  to  effectively 
implement  these  technologies.  By  replacing,  to  the  extent  possible,  the  detection/cued  survey 
sequence  with  a  one-pass  detection  and  classification  survey,  it  may  be  possible  to  extend  the 
feasibility  of  applying  classification  to  a  broader  range  of  sites. 
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2.0  TECHNOLOGY 


2.1  TECHNOLOGY  DESCRIPTION 

The  OPTEMA  sensor  is  built  around  the  G&G  Scienees  National  Instruments  (Nl)-based  data 
acquisition  framework.  These  data  acquisition  components  are  similar  to  those  incorporated  in  the 
first  generation  commercial  MetalMapper  systems.  Data  acquisition  hardware  is  housed  in  a 
National  Instruments  PXI-I042  chassis  and  includes  an  NI  PXI-8108  embedded  controller 
(Windows  OS)  and  six  8-channel  16-bit  NI  PXI-6I43  A/D  cards.  Intermediate  hardware  is  housed 
in  an  external  module  and  includes  the  transmitter  controller  and  power  distribution  board,  and 
three  16-channel  receiver  boards.  These  components  along  with  a  2000  Watt  inverter  are  contained 
in  a  ruggedized  vehicle-mounted  chassis  (Figure  1)  and  compose  the  OPTEMA  sensor  electronics. 


Tx/Rx  Module 

NIPXI-1042 

Chassis 

Inverter 


Operator  PC 


Figure  1.  The  OPTEMA  Sensor  Electronics  Include  Transmitter/Receiver  Boards, 
Analog-to-digital  Data  Acquisition  Hardware,  Embedded  Controller,  Inverter,  and 

Operator  PC. 

The  OPTEMA  sensor  head  comprises  five  transmitters  and  fourteen  3-axis  receivers  across  a  1.8 
meter  sensor  swath  (Figure  2).  This  design  ensures  that  three  orthogonal  magnetic  fields  are 
produced  at  any  across  track  location.  The  distribution  of  the  14  receivers  also  ensures  that  fields 
scattered  by  any  target  located  across  the  sensor  swath  will  be  characterized  sufficiently  to 
constrain  inversion  of  the  data. 
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Figure  2.  OPTEMA  Sensor  Head  Configuration. 

The  sensor  head  is  configured  to  provide  high  quality  classification  for  targets  encountered  anywhere 

across  the  entire  sensor  swath. 

The  transmitter  coils  include  four  horizontal  axis  transmitters  and  one  large  vertical  axis 
transmitter.  The  horizontal  axis  transmitters  are  wired  in  series  pairs  to  provide  two  effective 
orthogonal  excitation  axes.  The  horizontal  axis  transmitters  and  receiver  cubes  share  the  same 
reference  coordinate  frame  that  is  rotated  45  degrees  from  the  principal  coordinate  system  (i.e., 
referenced  to  the  direction  of  travel). 

After  completing  a  transect  (or  transects)  the  OPTEMA  data  can  be  gridded  to  form  a  detection 
map.  Several  options  exist  for  detection.  Early,  middle,  or  late  time  gates  within  the  Z-,  Y-,  or  X- 
component  data  can  be  used  to  generate  a  detection  map.  Subsequently,  Regions  of  Interest  (ROIs) 
are  selected  from  the  map  based  on  a  detection  threshold  value.  Eigure  3  shows  a  detection  map 
generated  from  data  collected  over  one  of  the  test  lanes. 

Once  threshold  analysis  is  complete;  classification  is  performed  on  each  ROI  extracted  from  the 
data  grid.  Each  ROI  comprises  a  number  of  EMI  soundings.  Depending  on  the  target  size,  target 
depth,  and  data  sample  resolution  a  typical  ROI  will  comprise  25-50  useful  soundings.  Thus,  a 
very  large  volume  of  data  exists  for  classification.  Once  a  subset  of  soundings  is  selected  from  the 
ROI,  the  data  is  inverted  in  each  sounding  to  produce  several  sets  of  polarizabilities  that  may  be 
analyzed  in  “clusters”  that  correspond  to  the  anomaly  source.  Eigure  4  shows  polarizabilities 
recovered  from  one  of  the  soundings  associated  with  the  37mm  ROI.  A  classification  decision  can 
be  made  by  matching  these  polarizabilities  to  those  of  known  targets  of  interest  catalogued  in  a 
TOI  library. 
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Figure  3. 


Z-Component  Detection  Map 


-1  0  1 

False  Easting  (m) 

OPTEMA  Z-  component  (Transmit  Z-,  Receive  Z-)  Detection  Map 
Generated  from  Test  Lane  Data. 


This  lane  contains  three  targets  (ISO  medium,  81  mm  mortar,  and  37mm  projectile)  all  buried  at  15  cm 
depth.  ROIs  corresponding  to  each  target  are  circled  in  red. 


Figure  4.  Polarizabilities  Recovered  from  Inversion  of  the  First  ROI. 

Blue,  red,  and  green  lines  represent  the  primary,  secondary,  and  tertiary  (respectively)  polarizabilities 
recovered  from  the  ROI  dynamic  data.  Grey  lines  represent  library  polarizabilities  generated  from  static 
calibration  data.  The  results  indicate  a  good  match  to  the  37mm  library. 
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2.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 


The  primary  advantage  of  the  OPTEMA  technology  over  existing  cued  EMI  sensors  is  the  ability 
to  acquire  effective  classification  data  in  one-pass  dynamic  mode.  This  capability  means  that  the 
OPTEMA  could  deliver  detection  and  classification  in  what  would  effectively  be  a  DGM  survey 
mode,  thus  obviating  the  need  for  a  secondary  cued  survey.  By  eliminating  the  cued  survey  from 
the  classification  work  flow,  the  one-pass  method  would  also  eliminate  the  ambiguity  associated 
with  reconciling  cued  EMI  data  with  dynamic  (or  DGM)  data.  Ultimately  this  approach  could  lead 
to  higher  confidence  in  classification  decisions  at  reduced  project  costs. 

The  main  limitation  of  the  current  technology  is  the  reduced  data  Signal  to  Noise  Ratio  (SNR) 
compared  to  that  of  the  advanced  sensors  (e.g.,  MetalMapper)  operated  in  cued  mode.  This  lower 
SNR  is  due  to  the  data  acquisition  parameter  constraints  imposed  by  the  dynamic  survey  mode 
(i.e.,  reduced  stacking).  These  constraints  are  fundamental  to  dynamic  surveys  and  will  need  to  be 
overcome  through  optimal  selection  of  data  processing  and  inversion  methodologies  as  described 
in  this  report. 

The  challenges  associated  with  reduced  data  SNR  during  dynamic  surveys  can  be  overcome  with 
effective  processing  strategies.  Results  from  the  initial  validation  tests  at  APG  did  not  reveal  any 
significant  limitations  in  detection  or  classification  performance  when  compared  to  performance 
of  other  advanced  cued  sensors  tested  in  the  same  areas  [1][3][4]. 
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3.0  PERFORMANCE  OBJECTIVES 

Performance  objectives  for  the  SWPG  demonstration  are  summarized  in  Table  1. 


Table  1.  Performance  Objectives. 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Results 

Quantitative  Performance  Objectives 

Detection  of  all 
TOIs 

Probability  of  detection 
(Prf)  of  all  seeded  and 
native  TOIs 

•  Dynamic  OPTEMA  data 

•  Ranked  anomaly  list 

•  Independent  scoring  report 

Prf  =1.0  for  all  TOIs 
encountered 

Pass 

Maximize  False 
Positive  Rejection 
(FPR)  Rate  at 
maximum  P^  value 

Number  of  non-TOIs 
correctly  classified  as  non- 
TOIs  out  of  total  number 
of  non-TOIs 

•  Dynamic  OPTEMA  data 

•  Ranked  anomaly  list 

•  Independent  scoring  report 

FPR>=0.70  for  all  TOIs 
detected  (i.e.,  at  least  70%  digs 
saved  with  maximum 
detection  of  TOIs) 

Pass 

Effective  stop  dig 
decision 

Number  of  TOIs  selected 
as  a  “dig”  out  of  the  total 
number  of  TOIs  on  the  list 

•  Ranked  anomaly  list 

•  Independent  scoring  report 

Selection  of  100%  of  TOIs  on 
the  list  as  digs  with 

FPR>=0.70 

Pass 

Maximize  correct 
classification  of 
TOIs 

Number  of  TOIs  classified 
as  correct  type  out  of  total 
number  of  TOIs 
encountered 

•  Dynamic  OPTEMA  data 

•  Ranked  anomaly  list 

•  Independent  scoring  report 

Probablility  of  Correct 
Classification  (Pcc)>=0.90 
(i.e.,  90%  or  greater  correct 
classification) 

Pass 

Accurate 
estimation  of 
target  locations 

Northing,  Easting,  and 
depth  mean  location  error 
(MLE) 

•  Dynamic  OPTEMA  data 

•  Estimated  target  locations 

•  Independent  scoring  report 

N,  E,  depth  MLE  <=.10m 

Pass 

Precise  estimation 
of  target  locations 

Northing,  Easting,  and 
depth  error  standard 
deviation  (SDV) 

•  Dynamic  OPTEMA  data 

•  Estimated  target  locations 

•  Independent  scoring  report 

N,  E,  SDV<=.10m 

Depth  SDV  <=.20m 

Pass 

Quantitative  Survey  Objectives 

Production  rate 

Effective  area  surveyed 
per  hour  of  operation 

•  Eield  logs,  OPTEMA  data 
file  time  stamps 

>=0.5acre/hour  (hr) 

Pass 

Area  coverage 

Effective  area  covered  by 
the  array 

•  GPS  and  IMU  data  from 
OPTEMA  data  files 

=100%  coverage  at  1.2m  line 
spacing 

Pass 

Along  track 
sample  separation 

Distance  between 
consecutive  EMI  samples 

•  GPS  and  IMU  data  from 
OPTEMA  data  files 

Mean  sample  distance 
throughout  each  survey  line 
<=.08m 

Max.  sample  distance  <=.15m 
SDV  of  sample  distance 
<=.02m 

Pass 

Instrument 
Verification  Strip 
(IVS)  survey 
position  accuracy 

Northing,  Easting,  and 
depth  location  error 

•  Dynamic  OPTEMA  data 
files  collected  daily  over  IVS 

•  Estimated  target  locations 

N,  E,  depth  maximum  location 
error  <=.15m 

Pass 

IVS  survey 

polarizability 

accuracy 

IVS  target  polarizability 
match  to  library 

•  Dynamic  OPTEMA  data 
files  collected  daily  over  IVS 

•  Recovered  target 
polarizabilities 

>=95%  match  to  library 

Pass 

Qualitative  Performance  Objectives 

Ease  of  use 

•  Operator  feedback 
regarding  ease  of  operation 
(e.g.,  navigation,  speed 
control,  etc.) 

NA 
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3.1  OBJECTIVE:  DETECTION  OF  ALL  TARGETS  OF  INTEREST 


The  objective  is  to  detect  all  seeded  and  native  TOIs  encountered  for  data  collected  during  the 
dynamic  survey.  This  test  establishes  the  OPTEMA’s  ability  to  function  as  a  detection  sensor. 

Data  Requirements 

Detections  will  be  identified  from  the  OPTEMA  data  acquired  during  the  dynamic  survey.  A 
threshold  based  on  the  most  difficult  target  scenarios  that  are  specific  to  the  SWPG  demonstration 
site  will  be  applied  (see  Section  6.2  Detection  Processing  for  details  on  detection  threshold 
selection).  After  applying  dipole  model  fitting  to  each  ROI,  a  list  of  target  locations  that  correspond 
to  anomaly  sources  will  be  generated.  Independent  scoring  analysis  of  the  target  list  will  determine 
detection  performance. 

Metric 

This  objective  applies  a  Probability  of  Detection  (Pd)  metric  to  assess  performance.  Pd  values  for 
each  area  will  be  generated  by  taking  the  ratio  of  the  total  number  of  TOIs  detected  to  the  total 
number  of  TOIs  encountered  by  the  OPTEMA  (as  determined  by  ground  truth). 

Success  Criteria 

A  Pd  =1.0  (100%  of  TOIs  detected)  will  be  considered  successful. 

3.2  OBJECTIVE:  MAXIMIZE  FALSE  POSITIVE  REJECTION  RATE  (FPR) 

The  objective  is  to  maximize  rejection  of  false  positives  (i.e.,  unnecessary  digs)  while  maintaining 
correct  identification  of  all  TOIs  detected  (i.e.,  with  Pd  value  maximized).  This  metric  will  be  used 
to  assess  discrimination  performance.  Accordingly,  the  false  positive  rejection  rate  at  the 
maximum  Pd  level  achieved  will  be  determined.  In  other  words,  if  there  is  a  missed  TOI  detection 
(i.e.,  Pd  <  100%),  the  FPR  rate  based  on  the  total  number  of  TOIs  detected  (i.e.,  the  total  number 
of  TOIs  presented  in  the  list)  will  still  be  determined.  If  a  TOI  is  present  on  the  list,  but  is  rated 
with  a  “no-dig”  decision,  the  FPR  rate  would  be  assessed  only  for  non-TOI  ranked  lower  than  this 
item.  This  approach  ensures  that  potential  discrimination  performance  can  be  assessed 
independently  of  detection  performance  or  analyst  confidence. 

Data  Requirements 

The  classification  to  all  targets  identified  from  the  ROI  detections  generated  from  the  dynamic 
survey  data  will  be  applied.  Each  ROI  will  contain  a  subset  of  soundings  from  the  dynamic  data 
that  corresponded  to  the  anomaly  source(s).  Based  on  the  results  of  inverting  these  data,  a  ranked 
list  of  likely  TOIs  will  be  generated.  Highest  confidence  TOIs  will  be  placed  at  the  top  of  the  list 
and  highest  confidence  non-TOIs  (i.e.,  clutter)  at  the  bottom  of  the  list.  The  objective  will  be  to 
rank  the  list  such  that  the  last  TOI  identified  on  the  list  would  be  ranked  above  at  least  70%  of  the 
non-TOIs.  This  list  will  be  submitted  for  independent  ground  truth  scoring. 

Metric 

This  objective  applies  a  False  Positive  Rejection  (FPR)  rate.  FPR  values  will  be  generated  by 
taking  the  ratio  of  the  number  of  non-TOIs  ranked  below  the  last  TOI  on  the  list  (regardless  of  dig 
decision  for  that  last  TOI)  out  of  the  total  number  of  non-TOIs  on  the  list. 
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Success  Criteria 


An  FPR  value  >=0.70  (70%  clutter  rejection)  will  be  considered  successful.  It  should  be  noted  that 
this  value  would  include  any  non-TOI  that  are  placed  in  a  “can’t  analyze”  category  (which  would 
mean  these  items  would  be  ranked  above  the  last  TOI).  This  objective  FPR  value  is  based  on  cost 
estimates  described  in  [5]  [6]  and  may  be  adjusted  depending  on  the  site  characteristics  (i.e., 
anomaly  density)  and  relative  performances  of  other  classification  sensors  used  at  the  site. 

3.3  OBJECTIVE:  EFFECTIVE  STOP  DIG  DECISION 

The  stop  dig  point  provides  a  good  assessment  of  practical  implications  for  achieving  effective 
classification.  The  stop  dig  point  indicates  the  analyst’s  overall  confidence  in  the  system. 

Data  Requirements 

After  ranking  the  anomalies  on  the  list,  a  stop  dig  point  will  be  selected.  Since  all  anomalies  will 
be  assigned  a  dig  or  no-dig  decision,  the  last  anomaly  to  be  assigned  a  dig  decision  will  represent 
the  stop  dig  point. 

Metric 

The  dig  decision  metric  will  divide  the  number  of  TOIs  assigned  as  a  “dig”  out  of  the  total  number 
of  TOIs  on  the  list. 

Success  Criteria 

The  objective  will  be  to  select  100%  of  the  TOIs  on  the  list  as  a  “dig”  while  maintaining  an  FPR 
>=0.70. 


3.4  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TOIS 

The  objective  is  to  correctly  classify  TOIs  by  type. 

Data  Requirements 

Classification  to  all  targets  identified  from  the  ROI  detections  generated  from  the  dynamic  survey 
data  will  be  applied.  Each  ROI  will  contain  a  subset  of  soundings  from  the  dynamic  data  that 
corresponded  to  the  anomaly  source(s).  Based  on  the  results  of  inverting  these  data,  polarizability 
curves  corresponding  to  each  anomaly  source  will  be  generated.  To  classify  each  ranked  TOI  by 
type,  polarizabilities  to  those  of  known  TOIs  at  the  site  will  be  compared.  These  classifications 
will  be  recorded  on  the  ranked  anomaly  list  submitted  for  independent  ground  truth  scoring. 

Metric 

This  objective  applies  a  Probability  of  Correct  Classification  (Pec)  metric.  Pec  values  will  be 
generated  by  taking  the  ratio  of  the  number  of  TOIs  correctly  classified  by  type  out  of  the  total 
number  of  TOIs  encountered. 

Success  Criteria 

A  Pcc>=0.90  (90%  correct  classification)  will  be  considered  successful. 
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3.5  OBJECTIVE:  ACCURATE  AND  PRECISE  TARGET  LOCATION  ESTIMATION 


The  objective  is  to  provide  accurate  and  precise  estimates  of  TOI  x,  y,  and  z  locations  (i.e.,  Easting, 
Northing,  and  depth  positions). 

Data  Requirements 

Classification  to  all  targets  identified  from  the  ROI  detections  generated  from  the  dynamic  survey 
data  will  be  applied.  Each  ROI  will  contain  a  subset  of  soundings  from  the  dynamic  data  that 
corresponded  to  the  anomaly  source(s).  Based  on  the  results  of  inverting  these  data,  position  and 
orientation  estimates  for  each  anomaly  source,  including  those  corresponding  to  TOIs  will  be 
extracted.  These  position  estimates  will  be  compared  to  the  ground  truth  locations  of  all  TOIs. 

Metric 

Mean  Eocation  Error  (MEE)  and  location  error  Standard  Deviation  (SDV)  will  be  used  as  metrics 
to  gauge  TOI  localization  accuracy  and  precision.  The  MEE  value  will  be  calculated  by  taking  the 
mean  error  value  of  the  Northing,  Easting,  and  depth  location  estimates  for  all  TOIs.  The  SDV 
value  will  be  calculated  by  taking  the  standard  deviation  for  each  of  these  errors. 

Success  Criteria 

An  MEE<=.10m  for  the  Northing/Easting  and  depth  estimates,  and  an  SDV<=.10m  and  <=.20m 
for  the  Northing/Easting  and  depth  errors,  respectively,  will  be  considered  successful.  These  values 
are  well  within  the  detection  radius  used  for  scoring  as  well  as  the  search  radius  used  for  target 
reacquisition.  The  depth  error  SDV  bounds  are  increased  over  those  of  the  horizontal  location  error 
due  to  the  potential  for  ground  surface  standoff  variability  across  the  array.  Although  the  target 
location  estimates  are  equally  accurate  in  all  3  dimensions,  because  the  array  is  fairly  wide  (almost 
2m)  it  is  likely  that  ground  standoff  (i.e.,  vertical)  variation  across  the  array  would  be  greater  than 
any  horizontal  location  errors  from  the  GPS. 

3.6  OBJECTIVE:  PRODUCTION  RATE 

Production  rate  is  a  useful  indicator  of  the  overall  efficiency  of  the  one-pass  method.  It  provides  a 
quantifiable  metric  to  be  used  for  comparison  to  combined  production  rates  associated  with 
DGM/dynamic  surveys  and  cued  surveys. 

Data  Requirements 

To  determine  production  rate,  the  detection  maps  to  identify  the  total  survey  area  covered  by  the 
OPTEMA  system  will  be  used.  Eield  logs  and  file  time  stamps  to  determine  an  effective  survey 
time  will  be  utilized.  Production  time  will  include  time  required  to  collect  data  along  each  transect, 
time  required  to  maneuver  the  vehicle  around  obstacles  and  turn  the  vehicle  around  at  the  end  of 
each  line,  and  time  required  for  regular  equipment  checks  (e.g.,  battery  voltage  check,  operation 
parameter  adjustments,  etc.).  To  the  extent  possible,  the  time  spent  on  inadvertent  system  down¬ 
time  associated  with  delays  not  related  to  standard  production  operations  will  not  be  included  (e.g., 
sensor  malfunctions,  survey  vehicle  stalling,  etc.). 
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Metric 

Production  rate  will  be  calculated  based  on  the  average  area  surveyed  per  hour. 

Success  Criteria 

A  production  rate  of  0.5acre/hr  or  greater  will  be  considered  successful.  This  value  forms  the  basis 
for  the  survey  cost  assumptions  (see  section  3.2  above). 

3.7  OBJECTIVE:  AREA  COVERAGE 

The  objective  is  to  maximize  sensor  coverage  of  the  survey  area.  While  it  is  possible  to  detect  and 
classify  objects  outside  of  the  sensor  footprint,  full  coverage  of  the  survey  area  will  ensure  optimal 
performance. 

Data  Requirements 

To  determine  the  area  covered  by  the  array,  the  GPS  and  inertial  sensor  data  recorded  in  the 
dynamic  OPTEMA  data  files  will  be  used  to  calculate  the  sensor  position  and  orientation  on  the 
ground  during  each  sample  and,  therefore,  the  surface  area  covered  by  the  array  for  each  survey 
line.  It  can  then  be  determined  if  any  gaps  occurred  between  adjacent  survey  lines.  1.2m  line 
spacings  will  be  implemented.  With  the  1.8m  physical  width  of  the  array,  this  spacing  should 
provide  about  60cm  overlap  between  adjacent  lines,  allowing  for  +/-  30cm  line  following  error  for 
each  survey  line. 

Metric 

Area  coverage  will  be  determined  by  dividing  the  surface  area  covered  by  the  array  within  the 
survey  area  by  the  total  survey  area. 

Success  Criteria 

Area  coverage  of  100%  will  indicate  success. 

3.8  OBJECTIVE:  ALONG  TRACK  SAMPLE  SEPARATION 

The  objective  is  to  sample  the  survey  area  with  enough  spatial  resolution  to  detect  and  classify  all 
TOIs.  Because  dynamic  classification  performance  is  influenced  by  the  density  of  data  acquired, 
maximizing  the  number  of  “looks”  at  each  target  by  decreasing  the  along  track  sample  spacing 
will  ensure  optimal  classification  performance  is  achieved. 

Data  Requirements 

To  determine  the  along  track  sample  spacing,  the  GPS  and  inertial  sensor  data  recorded  in  the 
dynamic  OPTEMA  data  files  to  calculate  the  linear  distance  between  each  sounding  location  will 
be  utilized. 

Metric 

Eor  each  line,  the  distance  moved  between  each  consecutive  EMI  sample  will  be  calculated. 
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Success  Criteria 

The  target  sample  spacing  is  5  cm.  Therefore,  each  survey  line  will  require  a  mean  sample  distance 
<=8  cm,  a  maximum  sample  distance  (over  the  entire  line)  <=15  cm,  and  a  sample  distance 
standard  deviation  <=2  cm. 

3.9  OBJECTIVE:  IVS  SURVEY  POSITION  ACCURACY 

To  verify  instrument  functionality  regular  surveys  of  the  IVS  will  be  conducted.  One  metric  that 
indicates  basic  system  functionality  is  recovery  of  accurate  estimated  target  locations  (based  on 
inversion  of  dynamic  IVS  survey  data).  Recovery  of  accurate  target  location  estimates  ensures  that 
both  the  EMI  and  positioning  sensors  are  functioning  properly. 

Data  Requirements 

IVS  target  location  estimates  require  dynamic  OPTEMA  data  files  acquired  over  the  IVS  targets. 
These  files  can  be  processed  in  the  field  to  provide  estimates  of  anomaly  locations  within  the  IVS. 

Metric 

The  estimated  target  locations  will  be  compared  to  the  ground  truth  coordinates  for  each  IVS  item 
location  to  indicate  the  accuracy  of  these  location  estimates. 

Success  Criteria 

Eocation  estimates  that  are  within  15  cm  of  the  ground  truth  coordinates  will  indicate  that  system 
components  are  functioning  properly. 

3.10  OBJECTIVE:  IVS  SURVEY  POUARIZABIUITY  ACCURACY 

Another  metric  that  indicates  basic  system  functionality  is  recovery  of  accurate  polarizabilities 
(based  on  inversion  of  dynamic  IVS  survey  data)  for  targets  located  in  the  IVS.  Recovery  of 
accurate  classification  features,  such  as  polarizabilities,  is  a  good  indication  that  the  system  is 
functioning  properly. 

Data  Requirements 

Recovery  of  IVS  target  polarizabilities  will  require  dynamic  OPTEMA  data  files  acquired  over  the 
IVS  targets.  These  files  can  be  processed  in  the  field  to  provide  classification  features,  such  as 
polarizabilities,  for  each  anomaly  encountered  in  the  IVS. 

Metric 

Polarizabilities  recovered  from  each  IVS  target  encounter  will  be  compared  to  standard  libraries 
for  each  of  these  IVS  items.  A  mean  squared  difference  is  applied  to  produce  a  metric  that  indicates 
the  fit  quality  between  the  recovered  polarizabilities  and  the  library  polarizabilities. 

Success  Criteria 

A  fit  quality  of  >=95%  between  the  recovered  and  library  polarizabilities  for  each  IVS  target  will 
indicate  success. 
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4.0  SITE  DESCRIPTION 


The  live  site  demonstration  was  conducted  at  the  former  Southwestern  Proving  Ground  located 
near  Hope,  AR.  The  SWPG  was  built  in  1941  to  prepare  for  the  U.S.  engagement  in  World  War 
IT  Between  1942  and  1945,  the  proving  ground  served  for  testing  small  arms  ammunition,  20  to 
155  mm  projectiles,  mortars,  rockets,  grenades,  and  up  to  500-lb  bombs  [7].  Following  the  end  of 
World  War  II,  the  proving  ground  was  closed  and  surface  clearance  activities  were  performed  to 
remove  ordnance  prior  to  transferring  the  lands  back  to  private,  municipal,  and  state  owners.  Since 
the  initial  clearance,  however,  MEG  have  continually  surfaced  and  over  8,000  ordnance  items  have 
been  removed  from  private  property  located  on  the  former  proving  ground.  More  detailed 
information  regarding  the  SWPG  site  history  can  be  found  on  the  United  States  Army  Corps  of 
Engineers  (USAGE)  web  site^ 

4.1  SITE  SELECTION 

The  survey  area  was  a  4-acre  portion  of  Recovery  Eield  15  (RE-15),  which  is  located  in  a  part  of 
the  former  SWPG  that  has  been  repurposed  for  agricultural  use  (see  Eigure  5).  The  site  is  suitable 
for  towed  array  surveys  as  there  is  minimal  vegetation  (hayfield),  moderate  terrain  (mostly  flat  to 
rolling),  and  minimal  surface  disturbance  (periodic  agricultural  activity). 


Figure  5.  The  Survey  Area  (blue  square)  is  a  4  -Acre  Portion  of  Recovery  Field  15 
Located  within  the  Boundaries  of  the  Former  SWPG. 
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http://www.swl.usace.armv.i'niI/Missions/MilitarvMissions/FormerlvUsedDefenseSites.aspx 
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This  site  was  selected  for  its  surface  conditions  (Figure  6),  which  are  amenable  to  larger  towed 
systems,  and  its  anomaly  density  (previous  activities  at  this  site  have  indicated  that  it  provides  a 
fairly  high  anomaly  density  of  ~2000+/acre).  The  most  notable  feature  of  the  site  topography  was 
the  pockmarked  surface  resulting  from  the  daily  grazing  activities  of  a  nearby  herd  of  cattle.  These 
ground  conditions  did  not  impact  the  vehicle  advance  rate,  but  did  jostle  the  sled  noticeably 
throughout  the  survey. 


Figure  6.  OPTEMA  Dynamic  Survey  in  RF-15  of  the  Former  SWPG. 


4.2  TARGETS  OF  INTEREST 

Suspected  targets  of  interest  at  the  site  include:  20mm,  37mm,  40mm,  57mm,  75mm,  76mm, 
90mm,  105mm,  120mm,  and  155mm  projectiles,  as  well  as  81mm  mortars.  The  most  difficult 
target  scenarios  are  expected  to  be  20mm  projectiles  buried  at  6  inch  depths  and  37mm  projectiles 
buried  at  12  inch  depths.  Libraries  for  all  the  aforementioned  targets  were  included  in  the  analysis. 
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5.0  TEST  DESIGN 


The  field  component  of  the  demonstration  was  designed  to  provide  the  data  required  for  evaluating 
the  performance  objectives  described  in  section  3.  Details  of  the  field  test  are  described  in  the 
following  subsections. 

5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

The  OPTEMA  field  study  was  coordinated  with  other  advanced  sensor  survey  activities  within  the 
same  4-acre  portion  of  RE- 15  during  September  2015.  A  field  team  from  Tetra  Tech  was 
performing  dynamic  MetalMapper  and  cued  TEMTADS  data  collection  as  part  of  an  on-going  live 
site  demonstration  project  (MR-201423).  The  Tetra  Tech  project  provided  the  logistical  support 
for  the  multi-system  demonstration  in  RE- 15.  This  support  included  target  seeding,  IVS  setup,  and 
intrusive  investigation.  The  OPTEMA  survey  was  scheduled  over  a  6  day  period  that  coincided 
with  the  Tetra  Tech  team’s  ongoing  cued  TEMTADS  survey. 

The  OPTEMA  demonstration  comprised  site  mobilization,  sensor  calibration/verification,  a  one- 
pass  dynamic  OPTEMA  survey  conducted  over  the  4  acres  in  RE- 15,  in-field  data  quality  control 
(QC),  transect  recollects  (as  indicated  by  the  in-field  QC),  and  demobilization.  Initial  calibration 
activities  were  used  to  verify  instrument  functionality  following  site  mobilization,  develop  site 
specific  noise  thresholds,  and  to  generate  any  site  specific  TOI  libraries.  These  initial  test  activities 
included:  static  data  collection  over  calibration  items  to  generate  polarizability  libraries  for  items 
likely  to  be  encountered  in  the  survey  area,  and  dynamic  surveys  over  the  IVS  to  test  positional 
accuracy  and  instrument  functionality  as  well  as  to  determine  noise  characteristics.  Once  initial 
system  verification  was  complete,  the  team  conducted  a  dynamic  survey  of  the  4-acre  area.  Table 
2  presents  a  schedule  of  the  completed  field  activities. 

Table  2.  Schedule  of  SWPG  Field  Activities. 


Sensor  Calibration 


Dynamic  Survey 


In-Field  QC/Data  Recollect 


Demobilization 

5.2  SITE  PREPARATION 

Site  preparation  was  provided  by  the  Tetra  Tech  team  under  project  MR- 201423.  Preparation 
included  TOI  seeding,  IVS  installation,  and  control  point  surveys. 
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5.3  SYSTEM  SPECIFICATION 


The  OPTEMA  sensor  head  is  mounted  in  a  non-metallic  tow  sled.  The  sled  features  a  wooden  tow 
boom,  a  protective  skid  plate,  and  solid  rubber  tires.  In  flat  terrain,  the  sled  rolls  on  the  wheels; 
however,  the  skid  plate  provides  additional  support  in  more  challenging  terrain  conditions.  During 
the  SWPG  demonstration,  the  pockmarked  ground  surface  meant  that  some  portion  of  the  skid 
plate  on  the  bottom  of  the  sled  was  typically  in  contact  with  ground.  The  vehicle  tow  bar  puts  the 
leading  edge  of  the  OPTEMA  sensor  head  approximately  three  meters  behind  the  hitch  point  on 
the  vehicle.  Eigure  7  shows  a  picture  of  the  sensor  tow  sled  operating  in  RE- 15. 


Figure  7.  OPTEMA  Tow  Sled  at  SWPG. 

The  OPTEMA  incorporates  a  Trimble  Real  Time  Kinematic  Differential  GPS  for  sensor  head 
position  data.  A  Trimble  RIO  receiver  is  mounted  directly  above  the  center  of  the  sensor  head. 
Inertial  measurements  are  provided  by  a  Microstrain  3DM-GX3-25  orientation  sensor.  The  Euler 
angle  outputs  provide  pitch,  roll,  and  yaw  measurements  for  the  OPTEMA  sensor  head.  The  IMU 
is  co-located  with  the  GPS  receiver. 

5.4  CALIBRATION  ACTIVITIES 

Calibration  activities  at  the  SWPG  demonstration  site  consisted  of  both  initial  and  daily  calibration 
routines.  These  calibration  activities  included  basic  instrument  verification  measurements  such  as 
dynamic  noise  and  static  (spike)  tests,  as  well  as  dynamic  classification  of  objects  buried  in  the 
IVS.  Figure  8  shows  an  example  of  dynamic  IVS  data  results  from  a  daily  SWPG  IVS  survey. 
During  the  IVS  tests,  the  acquired  dynamic  data  over  IVS  targets  was  used  to  ensure  that  the 
polarizabilities  recovered  from  dynamic  data  were  consistent  on  a  day-to-day  basis.  Polarizabilities 
recovered  from  IVS  data  should  match  the  reference  libraries  with  95%  fit  for  dynamic  IVS  survey 
data.  Additionally,  estimated  target  location  coordinates  recovered  from  inversion  of  the  IVS  data 
should  be  within  15cm  of  the  ground  truth  coordinates. 
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Instrument  verification  surveys  were  conducted  at  least  twice  daily  (morning,  afternoon,  and 
typically  mid-day)  to  ensure  consistency  throughout  each  day  of  operation.  For  each  IVS  survey, 
two  dynamic  passes  over  the  IVS  were  performed.  The  first  pass  was  performed  with  the  left  side 
horizontal  transmitter  pair  centered  over  the  targets  and  the  second  pass  was  performed  with  the 
right  side  horizontal  transmitter  pair  centered  over  the  targets.  These  two  offsets  ensured  that  all 
receiver  channels  were  adequately  assessed.  For  each  pass,  the  polarizability  match  and  estimated 
location  for  each  target,  and  assessed  the  dynamic  noise  standard  deviation  was  determined. 


shotput 


57mm 


<u 

•o 

Q. 

E 

< 


u 

T3 


Q. 

E 

< 


Time 

37mm 


Time 

Global  Position  Accuracy 


<u 

■o 


Q. 

E 

< 


<u 

■o 


a. 

E 

< 


Time 

90mm 


Time 

Inverted  Depth  Accuracy 


-0.5  0  0.5 

Easting  Error  (m) 


X 

shotput 

Target  Depth  (mflepth  (mf 

rror  (m) 

X 

57mm 

T001 

0.15 

0.10 

-0.05 

X 

37mm 

X 

90mm 

T002 

0.30 

0.31 

0.01 

---- 

15  cm  Halo 

T003 

0.15 

0.10 

-0.05 

T005 

0.45 

0.43 

-0.02 

Figure  8.  Examples  of  Library  Matching  and  Estimated  Location  Accuracy  for 

Dynamic  IVS  Data  Polarizabilities. 

Because  the  dynamic  polarizabilities  are  noisier  than  those  from  static  measurements,  the  fit  metric  is 
adjusted  to  be  less  sensitive  to  model  noise  beyond  the  2ms  decay  period.  an  example  this  figure  shows 
polarizabilities  obtained  from  dynamic  survey  data  over  the  four  targets  in  the  SWPG  IVS.  These 
measurements  were  performed  at  least  twice  daily  to  ensure  consistent  data  quality. 


19 


5.5  DATA  COLLECTION  PROCEDURES 


Navigation  for  the  dynamic  survey  was  provided  by  WRT  navigation  software.  Line  segment  files 
were  generated  prior  to  conducting  the  survey.  Applying  the  grid  coordinates  for  the  4-acre  survey 
area,  transect  lines  for  the  one  pass  survey  were  generated.  For  the  SWPG  survey,  1.2m  spacing 
for  transect  lines  in  the  survey  was  used.  This  spacing  provided  sufficient  array  overlap  to 
minimize  the  potential  for  line  gaps. 

Throughout  the  survey,  it  was  ensured  that  all  EMI  survey  activities  maintained  the  highest  Real 
Time  Kinematic  (RTK)  GPS  quality  of  “RTK  Fixed”.  In-field  quality  checks  were  performed  of 
each  survey  line  for  along-track  sample  spacing  and  GPS  quality  to  identify  any  potential  drop  in 
positional  accuracy.  Lines  containing  any  data  acquired  without  RTK  Fixed  quality  would  be 
recollected. 

Regular  quality  checks  also  included  identification  of  any  discrepancies  in  receiver  channel  output 
that  might  indicate  sensor  faults.  Such  faults  could  be  identified  through  the  analysis  of  the  daily 
calibration  tests  and  dynamic  noise  tests  conducted  in  the  IVS.  Additionally,  in-field  quality  checks 
were  performed  of  data  from  each  line  collected  in  the  survey  area.  For  each  survey  line,  the  along 
track  sample  spacing,  the  across  track  line  spacing,  and  the  dynamic  noise  levels  acquired  over 
background  locations  were  assessed.  This  in-field  analysis  was  conducted  using  the  survey  quality 
metrics  described  in  Section  3.  This  procedure  ensured  that  any  survey  lines  containing  faulty  data, 
sample  errors,  low  GPS  quality,  or  line  gaps  would  be  recollected  before  demobilizing. 

5.6  VALIDATION 

Upon  completion  of  the  SWPG  data  collection,  detection  and  dig  lists  were  generated  using 
standard  UXO  live  site  demonstration  scoring  formats.  Because  of  the  high  anomaly  density, 
intrusive  investigation  of  the  entire  4-acre  area  could  not  be  performed  under  the  demonstration 
scope.  A  subset  of  3  focus  areas  was  identified  for  intrusive  investigation.  These  areas  contained 
a  total  of  2022  anomalies  that  were  intrusively  investigated.  Due  to  the  sequence  of  the  multi¬ 
system  demonstration,  target  picks  were  based  primarily  on  the  TEMTADS  picks  (from  a  dynamic 
MetalMapper  survey  performed  earlier  in  the  summer);  however,  a  list  was  provided  of  OPTEMA 
picks  for  the  largest  of  the  three  focus  areas  (Figure  9)  that  was  incorporated  in  the  intrusive 
investigation. 
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Figure  9.  The  Largest  of  the  Three  Suh-areas  was  the  Southeast  Corner  Focus  Area 
(highlighted  here  on  the  OPTEMA  detection  map). 

OPTEMA  target  picks  in  this  sub-area  were  submitted  for  intrusive  investigation. 

Ranked  anomaly  lists  for  the  three  focus  areas  were  generated.  Final  ranked  anomaly  lists  included 
confidence  rankings  of  anomalies  in  the  focus  areas  with  highest  confidence  TOI  rankings  at  the 
top  of  the  list  and  highest  confidence  non-TOI  rankings  at  the  bottom  of  the  list.  These  results  were 
submitted  to  the  Program  Office  for  subsequent  independent  scoring  by  the  Institute  for  Defense 
Analyses  (IDA). 
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6.0  DATA  ANALYSIS  AND  PRODUCTS 

OPTEMA  data  analysis  includes  preprocessing,  detection  processing,  and  classification 
processing  and  analysis  stages. 

6.1  PREPROCESSING 

A  transmitter  current  normalization  is  performed  on  all  OPTEMA  data  files  by  dividing  all  data 
channel  values  by  the  peak  current  value  corresponding  to  the  appropriate  transmitter  (e.g.. 
Transmit- Y/Receive-Z  data  channels  are  divided  by  the  peak  Y-transmitter  current).  This  process 
ensures  that  only  the  size  and  number  of  windings  (both  of  which  are  constant)  for  each  transmitter 
affect  the  response  measured.  Thus,  no  discrepancies  in  transmitter  current  (which  may  vary)  will 
influence  the  data. 

After  each  data  file  is  current-normalized,  a  background  subtraction  step  is  performed  by 
detrending  the  data  acquired  along  each  survey  line.  A  window  length  containing  N-samples  is 
selected  and  the  median  value  of  the  N  samples  contained  in  this  window  is  removed  from  the 
datum  centered  in  the  window.  The  window  is  then  moved  by  one  sample  and  this  process  is 
repeated  along  the  entire  line  for  each  receiver  channel. 

A  final  preprocessing  step  is  performed  to  smooth  the  data.  This  step  includes  low-pass  filtering 
of  each  data  channel  throughout  a  transect  line  to  reduce  point-to-point  jitter.  This  smoothing  step 
greatly  improves  SNR  of  the  data  for  classification. 

6.2  DETECTION  PROCESSING 

Detection  maps  are  created  by  processing  dynamic  data  files.  Detection  processing  requires  data 
gridding  for  each  transmit/receive  pairing.  GPS  latitude/longitude  values  are  converted  to 
Universal  Transverse  Mercator  (UTM)  coordinates  (Easting  /  Northing).  Each  sounding  in  a 
dynamic  file  is  associated  with  a  GPS  Easting  and  Northing  coordinate  and  a  measurement  of 
sensor  head  pitch,  roll,  and  yaw  angles.  Receiver  cube  positions  for  each  sounding  are  calculated 
by  applying  an  Euler  transformation  to  the  vector  from  the  center  of  the  GPS  antenna  to  the  center 
of  each  receiver  cube.  This  transformation  creates  an  Easting/Northing  position  for  each  receiver 
cube  for  each  sounding. 

Eor  each  principal  transmit/receive  pairing  (i.e.,  Transmit-Z/Receive-Z,  Transmit- Y/Receive-Y, 
Transmit-X/Receive-X),  a  data  value  is  generated  for  each  receiver  cube  by  summing  the  values 
of  the  time  gates  spanning  130  ps  to  566  ps  in  the  corresponding  data  channel.  These  data  values 
are  then  mapped  to  a  2-D  space  using  the  receiver  cube  locations.  Einally,  2-D  interpolation  is 
applied  to  generate  the  final  uniformly-spaced  X-,  Y-,  and  Z-  data  maps.  Eigure  10  shows  the  Z- 
data  map  for  the  4-acre  survey  area  in  RE15. 

A  peak  detection  algorithm  is  applied  to  the  Z-component  map  using  a  threshold  based  on  the  data 
noise  floor  standard  deviation  or  site-specific  TOI  detection  thresholds.  A  detection  radius  is  applied 
to  identify  the  ROI  surrounding  each  peak.  The  radius  size  is  based  on  the  local  gradient  associated 
with  the  peak  and  the  number  of  peaks  associated  with  an  anomaly  (1  peak  for  Z-component  data). 
Einally,  across  track  and  along  track  indices  are  generated  for  each  alarm  in  an  ROI.  These  indices 
correspond  to  the  receiver  cube  and  sounding  number  associated  with  each  alarm  and  provide  the 
initial  starting  parameters  for  the  inversion.  Each  ROI  is  saved  as  a  data  volume  in  .mat  format. 
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Figure  10.  OPTEMA  Z-component  Data  Map  for  the  4-acre  Area  in  RF15. 

For  the  SWPG  demonstration,  the  selected  detection  threshold  was  based  on  the  suspected  TOI’s 
for  the  site.  Prior  to  the  survey,  model-based  depth  response  curves  for  the  most  difficult 
orientations  for  each  TOI  were  generated  and  verified  the  response  curves  with  test  stand  data.  The 
detection  threshold  applied  to  the  SWPG  data  ensured  that  all  TOI  would  be  detected  at  the  10- 
12X  diameter  depth. 

6.3  CLASSIFICATION  PROCESSING 

Following  detection  processing,  all  ROIs  are  further  processed  for  classification  analysis.  Based 
on  the  across  track  and  along  track  indices  for  each  ROI  alarm,  a  subset  of  soundings  is  selected 
from  the  ROI  dynamic  data  for  inversion.  For  the  SWPG  data  processing,  all  soundings  acquired 
when  the  sensor  head  center  was  within  1  meter  of  the  ROI  alarm  peak  were  selected.  For  each  of 
these  soundings,  an  inversion  was  applied  to  fit  a  dipole-based  forward  model  to  the  data  using  a 
least  squares  method.  The  forward  model  accounts  for  the  sensor  array  geometry,  the  position  of 
the  array  relative  to  the  target,  and  the  target  physical  features,  which  are  modeled  as  a  set  of 
orthogonal  magnetic  dipoles  (i.e.,  polarizability  tensor).  Once  model  parameters  are  chosen  to 
minimize  the  error  between  the  data  and  the  model  output,  the  resulting  target  polarizabilities  are 
selected  for  feature  classification. 

Finally  a  clustering  step  is  performed  such  that  each  ROI  detection  is  associated  with  a  unique 
set  of  polarizabilities  generated  from  the  data  inversion.  Clusters  are  generated  by  grouping 
polarizabilities  that  have  similar  size  and  decay  characteristics  and  that  are  associated  with 
estimated  locations  within  a  confined  area  (e.g.,  within  a  15  cm  radius).  This  process  removes 
any  outlier  polarizabilities  that  do  not  present  features  resembling  those  of  a  nearby  group.  Each 
remaining  group  of  polarizabilities  is  assigned  to  an  ROI  detection  and  these  groups  are  used  to 
classify  and  rank  the  anomalies. 
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6.4  CLASSIFICATION  ANALYSIS 


After  classification  processing  is  complete,  principal  polarizabilities  for  each  ROI  detection  are 
analyzed  for  classification  of  features.  Automated  TOI  rankings  are  assigned  based  either  on 
polarizability  fits  to  TOI  libraries  or  on  specific  classification  features  derived  from  the 
polarizabilities.  For  the  SWPG  demonstration,  the  ratio  of  the  late  time  to  early  time  separation 
between  the  primary  and  tertiary  polarizabilities  as  the  ranking  feature  (larger  values  at  the  top  of 
the  list;  smaller  values  at  the  bottom  of  the  list)  was  used.  This  feature  was  used  in  lieu  of  a  library 
match  given  the  possibility  for  new  TOI  that  were  not  in  the  libraries.  For  each  ROI  detection,  the 
feature  is  derived  from  the  average  of  the  associated  polarizability  cluster. 

Once  the  initial  automated  ranking  stage  is  complete,  an  analyst  reviews  the  polarizability  clusters 
against  the  nearest  library  match  as  well  as  the  estimated  location  clusters  plotted  on  the  2D 
detection  map  (Figure  11). 

Based  on  the  analyst’s  review,  adjustments  to  the  initial  ranking  order  can  be  made  to  create  a  final 
ranked  anomaly  list.  Highest  confidence  TOI  rankings  are  assigned  to  anomalies  that  provide  good 
fits  (e.g.,  >90%  match)  to  library  TOI  for  all  three  polarizabilities.  Lower  confidence  TOI  rankings 
are  assigned  to  anomalies  that  provide  good  fits  (e.g.,  >90%  match)  to  library  TOI  for  the  primary 
polarizability  only. 

Non-TOI  rankings  are  assigned  to  anomalies  based  on  symmetry  features  (e.g.,  ratio  of  secondary 
to  tertiary  polarizabilities),  decay  features  (e.g.,  ratio  of  late  time  to  early  time  for  primary 
polarizability),  and  size  features  (e.g.,  primary  polarizability  sum  of  all  time  gates).  Lowest 
confidence  non-TOI  rankings  are  assigned  to  anomalies  showing  good  symmetry,  large  size,  and 
long  decay  features.  Highest  confidence  non-TOI  rankings  are  assigned  to  anomalies  showing  poor 
symmetry,  small  size,  and  short  decay  features. 


Figure  II.  Analyst’s  Review.  Polarizability  Clusters  (left:  blue,  red,  and  green  curves) 
are  Compared  to  the  Closest  Library  Match  (left:  20mm  shown,  grey  curves). 

The  detection  map  (right)  presents  the  associated  clusters  of  estimated  locations  (right:  blue  dots)  and 
average  estimated  location  (right:  magenta  circle)  for  the  anomaly. 
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For  the  final  classification  list,  categories  are  assigned  to  all  anomalies  using  the  standard  ESTCP 
ranked  anomaly  format:  -1  -  training  set;  0  -  can’t  extract;  1  -  likely  TOI;  2  -  can’t  decide;  3  - 
likely  non-TOI.  Additionally,  a  dig  decision  is  provided  for  each  anomaly:  1  -  dig;  0  -  don’t  dig. 
This  dig  decision  defines  the  stop  dig  point.  Type  (i.e.,  size)  is  also  provided  for  any  item  listed  as 
a  TOI  (category  1). 

Finalizing  the  target  ranking  and  stop  dig  point  is  an  iterative  process  where  training  data  are 
requested  if  the  analyst  suspects  the  presence  of  potentially  new  TOI  types  specific  to  the  site. 
Once  the  analyst  is  confident  all  TOIs  have  been  identified  as  such  on  the  list,  the  dig  sheet  is 
submitted  for  independent  scoring.  For  this  demonstration,  two  requests  for  training  data  were 
submitted  before  submitting  the  final  ranked  anomaly  list. 
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7.0  PERFORMANCE  ASSESSMENT 


The  IDA-generated  Receiver  Operating  Characteristic  (ROC)  curve  for  scoring  of  the  2022  target 
picks  within  the  three  RF-15  focus  areas  is  presented  in  Figure  12.  This  curve  shows  the  OPTEMA 
classification  performance. 


OPTEMA 

SWPG2  WhiteRiver  AdvancedModels  OPTEMA  None  Custom  s2  v1 


No  Difficult  TOIs 


Figure  12.  OPTEMA  ROC  Curve  for  the  2022  Target  Picks  Identified  in  the  Three 

Focus  Areas  of  RF-15. 

The  orange  dot  represents  the  100%  TOI  classified  correctly  operating  point.  The  blue  dot  represents  the 
stop  dig  point.  The  red  line  corresponds  to  items  classified  as  UXO.  The  yellow  line  corresponds  to  items 
classified  as  low-confidence  non-UXO  (digs).  The  green  line  corresponds  to  items  classified  as  high- 
confidence  non-UXO  (no-digs).  The  black  dashed  line  corresponds  to  training  dig  requests. 

Overall,  these  results  indicate  that  correct  classification  of  100%  of  the  TOI  with  82%  clutter 
rejection  at  the  stop  dig  point  was  achieved.  At  the  100%  efficiency  operating  threshold  (i.e.,  the 
point  where  maximum  clutter  rejection  can  be  achieved  with  no  missed  TOI)  95%  clutter  rejection 
was  achieved.  Details  of  the  specific  performance  objectives  are  presented  in  the  following 
subsections. 
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7.1  OBJECTIVE:  DETECTION  OF  ALL  TARGETS  OF  INTEREST 


Of  the  2022  targets  identified  within  the  three  focus  areas,  29  were  TOI  (all  seeded).  Of  these  29 
TOI,  13  were  located  within  the  southeast  focus  area.  OPTEMA  detections  were  submitted  for  this 
southeast  focus  area  only  due  to  time  constraints  on  the  dig  team  activities  at  the  site.  For  each  of 
the  13  TOI  in  this  focus  area,  the  lateral  offset  to  the  closest  OPTEMA  detection  was  calculated. 
All  13  TOI  had  an  OPTEMA  detection  that  was  within  15  cm  of  the  ground  truth  location  (mean 
offset  error  =  5.0cm,  maximum  offset  error  =  11.0cm,  standard  deviation  of  error  =  3.4cm). 
Therefore,  the  OPTEMA  detected  100%  of  the  TOI,  which  made  it  possible  to  assess  detection 
performance. 

7.2  OBJECTIVE:  MAXIMIZE  FALSE  POSITIVE  REJECTION  RATE 

The  objective  was  to  reject  at  least  70%  of  the  clutter  encountered  at  the  site.  At  the  100%  TOI 
correctly  classified  operating  point,  95%  clutter  rejection  (104  clutter  items  dug  out  of  1993  total 
clutter)  was  achieved.  At  the  100%  classified  UXO  operating  point  (i.e.,  all  items  classified  as 
UXO  on  the  list),  94%  clutter  rejection  (122  clutter  items  dug  out  of  1993  total  clutter)  was 
achieved.  At  the  stop  dig  point,  82%  clutter  rejection  (353  clutter  items  dug  out  of  1993  total).  The 
OPTEMA  performance  results  are  summarized  in  Table  3. 


Table  3.  OPTEMA  Performance  Summary 


OPERATING  POINT 

CLUTTER  REJECTION  ACHIEVED 

100%  TOI  Dug 

95% 

All  Items  Classified  UXO  Dug 

94% 

Stop  Dig 

82% 

7.3  OBJECTIVE:  EFFECTIVE  STOP  DIG  DECISION 

One  of  the  challenges  of  this  site  was  the  large  amount  of  munitions  debris  (fragments)  that  were 
of  similar  size  to  the  20mm  projectile  TOI.  Determining  a  stop  dig  point  that  would  reject  the 
majority  of  these  clutter  items  without  producing  a  false  negative  required  several  training  digs  to 
confirm  the  selection  of  appropriate  ranking  criteria. 

The  initial  ranking  compared  the  mean  separation  between  the  primary  and  tertiary  polarizabilities 
at  early  time  (subsequently  referred  to  as  the  K1  parameter)  to  the  mean  separation  between  the 
primary  and  tertiary  polarizabilities  at  late  time  (subsequently  referred  to  as  the  K2  parameter)  for 
each  polarizability  cluster  associated  with  a  detected  anomaly.  After  this  initial  ranking,  an  analyst 
reviewed  the  polarizabity  cluster  associated  with  each  ranked  anomaly  against  TOI  libraries.  Items 
near  the  top  of  the  list  (within  the  first  400)  that  presented  polarizabilities  similar  to  those  of  known 
TOI  (particularly  the  20mm  projectile)  were  flagged  by  the  analyst  for  training  dig  requests  to 
confirm  whether  the  item  was  a  TOI  or  non-TOI  (Figure  13).  Additionally,  items  associated  with 
a  high  K2/K1  value  that  did  not  match  known  TOI  libraries  were  also  flagged  for  training  dig 
requests. 
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A  total  of  91  training  digs  (8  TOI,  83  clutter)  from  the  ranked  list  was  requested.  For  the  remaining 
ranked  anomalies,  the  stop  dig  point  was  set  at  230  anomalies  after  the  last  anomaly  classified  as 
a  UXO  (based  on  the  analyst’s  assessment  of  library  match).  Ground  truth  delivery  revealed  that 
this  stop  dig  deeision  was  effeetive  as  it  did  not  lead  to  any  false  negatives.  Additionally  it 
aeeomplished  the  objeetive  of  at  least  70%  elutter  rejeetion  (82%  elutter  rejeetion  aehieved).  Given 
the  large  amount  of  20mm-sized  debris,  it  is  believed  that  this  stop  dig  threshold  was  appropriately 
eonservative  to  reduee  the  possibility  of  a  missed  20mm  TOI. 


SW2-40  (20mm  TOI) 


^  Dod<».'.0^/V5o(5  ■ 

Ictn,  XD. 


SW2-2356  (Frag) 

I  V  ^(Jo)-32)5C> 

]>I6  Tv^pe  ;  l«t) 


LeMCtH-  II 


Figure  13.  Items  Selected  for  Training  included  those  that  Presented  Polarizahilities 

Similar  to  those  of  Known  TOI. 


Examples  include  SW2-40  (a  20mm  TOI)  and  SW2-2356  (a  20mm-sized  clutter  item).  SW2-40  was 
selected  to  confirm  the  expected  TOI  result.  SW2-2356  was  selected  to  confirm  that  the  slight  deviation 
from  the  20mm  library  in  early  time  could  be  used  reliably  to  distinguish  20mm-sized  debris  from  20mm 
TOI.  The  polarizability  clusters  associated  with  each  item  (blue,  red,  and  green  curves)  are  compared  to 

the  20mm  TOI  library  (grey  curves). 


7.4  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TOI 


All  29  TOI  were  eorreetly  classified  as  sueh  on  the  final  ranked  anomaly  list.  Additionally,  size 
estimates  for  whether  eaeh  TOI  diameter  was:  diam.  <  50mm,  50mm  <  diam.  <  lOOmm  were 
provided  eorreetly  for  all  TOI.  Size  estimates  were  based  on  the  library  mateh  observed  during  the 
analyst’s  review. 
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7.5  OBJECTIVE:  ACCURATE  AND  PRECISE  TARGET  LOCATION  ESTIMATION 


The  estimated  target  locations  (Northing,  Easting,  and  depth)  were  calculated  from  the  average 
values  recovered  from  the  polarizability  cluster  analysis  for  each  anomaly  (i.e.,  model  output). 
Using  this  mean  value  proved  to  be  an  effective  method  to  localize  each  source  location. 

Mean  error  values  and  standard  deviation  are  reported  in  Table  4  for  each  TOI  localization 
parameter  (i.e.,  Northing/Easting,  depth).  Errors  were  calculated  by  comparing  the  estimated  TOI 
locations  derived  from  the  polarizability  clusters  to  the  values  recorded  in  the  ground  truth  sheet, 
which  were  based  on  intrusive  investigation  results.  Eor  each  of  these  parameters,  the  objective 
values  were  within  10cm  for  the  mean  and  10cm  (Northing/Easting)  and  20cm  (Depth)  for  the 
standard  deviation. 

Table  4.  Mean  Location  Error  and  Standard  Deviation  for  TOI  Location  Estimates. 


RF-15  FOCUS  AREA  TOI 

Mean  Error  (cm) 

Standard 
Deviation  (cm) 

Maximum  Error 
(cm) 

N  orthing/Easting 

5.5 

2.4 

11.8 

Depth 

3.3 

4.6 

14.0 

7.6  OBJECTIVE:  PRODUCTION  RATE 

The  initial  survey  of  the  4-acre  area  was  completed  over  the  course  of  two  days.  The  first  day 
included  transport  of  the  equipment  to  the  field  area,  initial  calibration,  and  initial  IVS  testing. 
Therefore,  surveying  lines  was  not  started  until  the  middle  of  the  first  day. 

Data  was  collected  along  north/south  transects  that  were  approximately  130  meters  in  length. 
Transects  were  spaced  1.2  meters  apart,  resulting  in  a  total  of  110  lines  to  complete  the  4-acre 
survey  area.  The  surveying  utilized  a  race  track  pattern  (skipping  adjacent  lines  for  each  return 
path)  in  order  to  increase  the  turning  radius  at  the  end  of  each  transect.  This  procedure  helped  to 
minimize  time  spent  turning  the  vehicle  between  lines. 

Typical  survey  activities  included  a  morning  IVS  test,  a  mid-day  IVS  test  followed  by  a  break  to 
recharge  batteries,  and  an  end-of-day  IVS  test.  Time  stamps  from  the  data  files  indicate  that  three 
hours  was  spent  on  the  first  day  (following  the  initial  calibration  activities)  collecting  survey  data 
and  8  hours  on  the  second  day  to  complete  the  data  collection  of  the  4-acre  area.  Thus,  total  time 
collecting  data  was  approximately  11  hours;  however,  during  this  period  some  minor  data 
collection  issues  were  encountered  that  required  some  inadvertent  down  time.  These  issues 
included  a  cow  knocking  over  the  GPS  base  station,  a  broken  axle  on  the  tow  sled,  and 
implementing  a  different  GPS  rover  on  the  survey  vehicle.  When  factoring  in  these  incidents,  the 
actual  survey  time  for  the  4  acres  was  very  close  to  8  hours,  which  produces  the  objective  0.5 
acre/hr  production  rate. 
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Following  this  initial  survey,  data  was  recollected  over  portions  of  the  survey  area  before 
demobilization.  Some  of  these  recollects  were  a  result  of  the  aforementioned  data  collection  issues. 
Additional  data  recollects  were  performed  due  to  the  presence  of  a  large  east/west  swale  at  the 
north  end  of  the  area.  Several  lines  were  recollected  in  east/west  transects  along  the  swale  to  ensure 
that  the  sensor  maintained  closer  ground  standoff  in  these  areas.  Data  recollects  were  not  factored 
in  to  the  production  rate  calculation. 

7.7  OBJECTIVE:  AREA  COVERAGE 

Before  demobilization,  a  quality  check  was  performed  of  the  data  coverage  for  the  survey  area. 
All  data  files  were  processed  in  Matlab  to  determine  the  total  sensor  footprint  (in  UTM 
coordinates)  along  each  transect.  After  processing  all  files,  it  was  confirmed  that  there  were  no 
gaps  between  adjacent  lines  at  any  location  within  the  4-acre  area.  This  process  ensured,  prior  to 
departing,  that  100%  coverage  of  the  area  had  been  achieved. 

7.8  OBJECTIVE:  ALONG  TRACK  SAMPLE  SEPARATION 

To  ensure  sufficient  sample  density,  an  in-field  QC  software  tool  was  implemented  that  enabled 
quality  checks  of  data  from  each  survey  line.  For  each  data  line  collected,  a  review  was  performed 
of  the  average  along  track  sample  spacing  (objective:  <  8cm),  the  maximum  sample  spacing 
(objective:  <  15cm),  and  the  standard  deviation  of  the  sample  spacing  (objective:  <  2cm).  This 
process  ensured  that  all  data  met  the  along  track  sample  objectives. 

7.9  OBJECTIVE:  IVS  SURVEY  POSITION  ACCURACY 

IVS  tests  were  performed  three  times  each  day:  prior  to  the  day’s  survey  activities,  mid-day  during 
a  battery  change,  and  at  the  end  of  the  day.  For  each  IVS  target,  the  in-field  QC  software  processed 
the  relevant  data  to  produce  polarizabilities  and  an  estimated  target  location.  This  processing  step 
was  performed  immediately  after  collecting  IVS  data,  allowing  the  operators  to  view  results  in 
real-time  to  assess  instrument  functionality.  The  relevant  information  provided  to  the  operator  is 
shown  in  Figure  8.  In-field  quality  checks  included  observing  the  offset  between  the  estimated 
location  for  each  IVS  target  and  the  corresponding  ground  truth  location.  An  offset  <15cm  was 
required  for  passing  the  IVS  quality  check.  Throughout  the  survey,  all  IVS  tests  performed  passed 
this  position  accuracy  test. 

7.10  OBJECTIVE:  IVS  SURVEY  POLARIZABILITY  ACCURACY 

In  addition  to  target  location  estimates,  the  in-field  IVS  check  also  provided  a  library  match  metric 
for  each  set  of  IVS  target  polarizabilities  (Figure  8).  A  match  of  95%  or  better  was  required  for 
passing  this  quality  check.  All  IVS  tests  performed  passed  this  polarizability  match  test  for  all  IVS 
targets. 

7.11  QUALITATIVE  PERFORMANCE  OBJECTIVES 

From  a  user’s  perspective,  dynamic  classification  survey  procedures  using  a  tow  sensor  such  as 
the  OPTEMA  are  very  similar  to  those  of  standard  DGM  surveys.  Consequently,  many  of  the 
sensor  positioning  requirements  that  create  challenges  during  cued  surveys  are  not  an  issue. 
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The  main  operational  requirements  for  the  OPTEMA  include  basic  line  following  in  the  survey 
area  such  that  complete  coverage  is  achieved  and  periodic  (i.e.,  beginning  and  end  of  the  day) 
calibration  routines  to  ensure  instrument  functionality.  Some  of  the  more  onerous  tasks  associated 
with  cued  surveys,  such  as  frequent  background  data  collection  and  extensive  maneuvering  to 
center  the  sensor  over  a  set  of  2-D  coordinates  are  not  required.  From  this  perspective,  the 
OPTEMA  provides  a  straightforward  solution  for  classification-level  surveys  and  offers  some 
distinct  advantages  over  the  cued  approach. 

During  the  SWPG  demonstration,  the  in-field  quality  checks  were  straightforward  to  implement 
and  enabled  the  identification  of  any  data  quality  problems  and  subsequent  corrective  actions  (i.e., 
recollects)  before  demobilization.  The  quality  checks  included  the  IVS  test,  which  evaluated 
location  accuracy  and  polarizibility  accuracy,  and  the  data  QC  test,  which  evaluated  dynamic  noise 
profile  and  along  track  sample  spacing  for  each  survey  line.  The  efficiency  of  the  dynamic  survey 
operations  coupled  with  the  effective  in-field  quality  checks  enabled  the  field  team  to  mobilize, 
perform  the  survey,  and  demobilize  within  a  period  of  6  days. 
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8.0  COST  ASSESSMENT 


The  primary  cost  benefit  of  the  OPTEMA  is  the  removal  of  the  cued  survey  from  the  classification 
workflow.  The  costs  of  the  one-pass  classification  method  would  be  comparable  to  those  of  the 
remaining  portions  of  the  existing  workflow.  The  OPTEMA  survey  itself  is  similar  to  the  standard 
DGM  survey  conducted  as  part  of  production  operations.  The  0.5  acre/hr  production  may  be 
slightly  lower  than  the  rate  achieved  using  conventional  DGM  arrays.  Requirements  for  field 
personnel  are  also  similar  to  those  for  standard  DGM  surveys  (i.e.,  two  technicians  for  vehicle- 
towed  surveys). 

Regarding  the  costs  associated  with  the  post-survey  data  analysis,  it  is  believed  that  costs 
associated  with  analysis  of  OPTEMA  data  would  be  comparable  to  the  combined  costs  of  the  target 
picking  and  classification  analysis  stages  currently  required  for  cued  classification.  The  one-pass 
and  cued  methods  both  require  target  detection  and  target  classification  stages  as  part  of  the 
analysis.  Time  and  personnel  requirements  for  both  methods  are  similar  (i.e.,  review  of  each  target 
by  a  trained  analyst).  Processing  of  OPTEMA  data  is  more  computationally  intensive  due  to  the 
increased  volume  of  data;  however,  any  associated  costs  due  to  the  higher  data  density  would  be 
negligible  since  any  increased  burden  is  placed  on  processing  hardware,  not  on  personnel. 

8.1  COST  MODEL 

Table  5  provides  a  summary  of  the  cost  elements  associated  with  an  OPTEMA  survey. 

Table  5.  Cost  Requirements  for  OPTEMA  Survey. 


Cost  Element 

Data  Tracked  During  Demonstration 

Cost 

Mobilization  and 
demobilization 

•  Shipment  to  and  from  site 

•  Labor  required  to  pack  and  prep  equipment  (2  field 
technicians) 

•  Air  travel  for  3  field  personnel 

$28,811 

Site  preparation 

•  Labor  for  equipment  assembly  on-site,  initial  calibration 
and  IVS  test  (2  field  technicians,  1  project  geophysicist) 

•  Per  diem  rates  (3  personnel,  1  day) 

•  Equipment  rental  (survey  vehicle  and  GPS,  1  day) 

$5,929 

Survey  costs 

•  Labor  for  0.5  acre/hr  production  rate  (2  field  technicians) 

•  Labor  for  on-site  QC  (project  geophysicist) 

•  Labor  for  back-office  QC  support  as  needed 

•  Per  diem  rates  (3  personnel) 

•  Equipment  rental  (survey  vehicle  and  GPS) 

$1,941  per  acre 

(3  acre/day  production) 

Detection  data 
analysis  costs 

•  Labor  for  target  picking  analysis  of  OPTEMA  data  ( 1 
data  analyst  and  1  QC  geophysicist) 

•  Labor  for  project  management 

$409  per  acre 

Classification  data 
analysis  costs 

•  Labor  for  classification  analysis  of  OPTEMA  data  (1 
data  analyst,  analysis  of  400  anomalies/day) 

•  Labor  for  final  dig  list  review  (project  geophysicist, 
review  and  QC  geophysicist  final  review) 

•  Labor  for  project  management 

$4.81  per  anomaly 
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8.1.1  Mobilization  and  Demobilization 


Mobilization  costs  include  shipping  costs  to  transport  equipment  to  and  from  the  site,  labor 
required  to  disassemble,  prepare,  and  pack  equipment,  and  travel  costs  for  the  field  team.  The 
OPTEMA  system  is  shipped  in  two  large  crates  (4’  x  4’  x  8’)  for  the  sensor  head  and  tow  sled 
parts,  and  two  smaller  crates  (3’  x  3’  x  4’)  for  the  electronics,  spares,  and  survey  tools.  All  crates 
can  be  maneuvered  and  loaded  on  to  a  lift-gate  equipped  box  truck  using  a  pallet  jack  so  delivery 
to  remote  sites  is  possible  (i.e.,  those  with  no  formal  receiving  facilities  as  was  the  case  for  the 
SWPG  demonstration).  Preparation  requires  approximately  1  week  for  two  field  technicians. 
Additional  costs  include  materials  for  sled  reinforcement  and  spares  (e.g.,  fiberglass  parts). 

8.1.2  Site  Preparation 

Site  preparation  includes  the  assembly  of  the  system  upon  arrival,  initial  function  and  system 
verification  tests,  and  related  on-site  QC  analysis.  Associated  labor  includes  1  full  day  for  2  field 
technicians  and  an  on-site  geophysicist.  Additional  costs  include  per  diem  for  3  personnel  and 
equipment  rental  (GPS  and  survey  vehicle). 

8.1.3  Survey  Costs 

Survey  costs  include  per  diem  and  labor  for  2  field  technicians  and  1  on-site  geophysicist,  as  well 
as  labor  for  back-office  quality  support  by  a  data  analyst.  A  production  rate  of  0.5  acre/hr  (3 
acre/day)  is  assumed.  Additionally,  for  each  day  of  data  collection,  2  hours  of  back-office  support 
are  required.  Additional  costs  include  equipment  rental  (GPS  and  survey  vehicle). 

8.1.4  Detection  Data  Analysis  Costs 

Detection  analysis  is  performed  off-site  so  is  not  associated  with  any  travel  or  per  diem  costs.  Most 
of  the  detection  processing  is  automated,  but  some  analyst  oversight  is  required  including  the 
selection  of  appropriate  detection  thresholds,  and  quality  review  of  the  detection  map  (i.e.,  visual 
confirmation  that  the  thresholds  are  implemented  properly  to  identify  anomalies  on  the  map). 
Associated  labor  costs  for  this  effort  include  one  day  for  a  data  analyst  and  four  hours  for  a  QC 
geophysicist  for  4  acres  of  data  collected.  Additional  costs  include  labor  for  project  management. 

8.1.5  Classification  Data  Analysis  Costs 

Classification  analysis  is  also  performed  off-site.  These  costs  include  labor  for  a  data  analyst, 
project  geophysicist,  QC  geophysicist,  and  project/technical  manager.  Initial  classification 
analysis  includes  visual  review  by  the  analyst  of  classification  features  associated  with  each 
polarizability  cluster  (i.e.,  each  anomaly).  The  analyst  reviews  the  polarizabilities  against  the  TOI 
libraries,  identifies  any  potential  new  TOI,  and  modifies  the  ranking  order  appropriately.  The 
experience  with  this  demonstration  indicated  that  it  is  possible  for  an  analyst  to  review 
approximately  400  anomalies  per  day  during  this  stage.  Final  classification  includes  a  review  by 
the  project  geophysicist  of  the  analyst’s  final  ranked  anomaly  list  followed  by  subsequent  review 
by  the  QC  geophysicist  for  QC  seed  identification.  Labor  for  this  final  analysis  cost  includes  1  day 
for  the  project  geophysicist  and  4  hours  for  the  QC  geophysicist. 
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8.1.6  Overall  Cost  Analysis 

Here  an  example  is  provided  to  demonstrate  the  potential  cost  savings  that  could  be  achieved  by 
implementing  an  OPTEMA  survey.  Consider  a  100-acre  site  with  an  anomaly  density  of  250 
anomalies/acre.  A  few  basic  cost  assumptions  can  be  applied  using  data  from  the  2014  ESTCP 
Spencer  Range  summary  report  [5].  It  is  assumed  that  combined  field  survey  and  analysis  rates  of 
$  1000/acre  for  an  EM-61  DGM  survey  and  $30/anomaly  for  a  MetalMapper  cued  survey.  At  the 
aforementioned  250  anomalies/acre  density,  this  creates  a  total  survey /analysis  rate  of  $8500/acre 
or  $850,000  total  for  the  site. 

To  determine  an  estimated  cost  for  using  the  OPTEMA  at  this  site,  the  aforementioned  cost 
elements  can  be  applied.  These  include  a  total  survey  and  target  identification  analysis  cost  of: 

($1941/acre  -i-  $409/acre)  x  100  acres  =  $235,000 
Classification  analysis  costs  are  based  on  cost  per  anomaly  and  would  be: 

$4. 81/anomaly  x  250  anomalies/acre  x  100  acres  =  $120,250 
This  brings  the  total  cost  to: 

$235,000  -t  $120,250  =  $355,250 

Eor  the  aforementioned  site,  there  is  a  total  cost  of  $355,250  or  an  approximate  58%  reduction  in 
total  survey/analysis  costs  when  compared  to  the  cued  classification  approach.  The  cost  savings 
becomes  even  more  significant  when  anomaly  densities  are  higher;  for  example  if  the  anomaly 
density  for  the  same  site  were  600  anomalies/acre.  Eor  the  DGM/cued  approach  survey  costs  would 
be: 


($  1000/acre  x  lOOacres)  -i-  ($30/anomaly  x  600  anomalies/acre  x  100  acres)  =  $1,900,000 
OPTEMA  costs  for  the  same  site  would  be: 

($2350/acre  x  lOOacres)  -i-  ($4.81 /anomaly  x  600  anomalies/acre  x  100  acres)  =  $523,600 

In  this  case,  the  OPTEMA  would  provide  an  approximate  72%  cost  savings,  which  emphasizes 
the  increased  cost  benefit  at  higher  density  sites. 

8.2  COST  DRIVERS 

While  the  one-pass  detection/classification  method  could  provide  significant  cost  savings  at  any  site, 
the  greatest  returns  for  OPTEMA  deployments  would  be  realized  in  large  open  field  areas  and  high 
density  sites.  Highest  production  rates  would  be  achieved  in  areas  amenable  to  vehicle-towed 
surveys,  particularly  those  sites  that  allow  for  long,  straight  survey  transects.  Additionally,  sites  that 
contain  higher  anomaly  densities  would  also  realize  significant  cost  savings.  Because  cued  survey 
costs  scale  considerably  with  anomaly  density  (i.e.,  production  rates  for  cued  systems  are  generally 
given  in  anomalies/hr  rather  than  acres/hr),  high  density  areas  significantly  drive  up  cued  survey 
costs,  but  have  no  significant  effect  on  OPTEMA  production. 
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9.0  IMPLEMENTATION  ISSUES 


Over  the  course  of  the  survey,  a  few  data  collection  issues  were  encountered  that  required 
corrective  actions.  The  first  incident  occurred  at  the  beginning  of  data  collection  over  the  first  few 
survey  lines.  Initially  the  GPS  rover  on  the  tow  sled  was  used  to  provide  the  position  data  for  the 
navigation  software;  however,  after  the  first  few  lines  it  was  quickly  realized  that  the  offset 
between  the  sled  and  the  vehicle  made  it  difficult  to  precisely  follow  the  survey  lines  using  the 
navigation  software.  This  issue  was  quickly  resolved  by  incorporating  a  second  GPS  rover  into  the 
survey  operations.  The  additional  rover  was  placed  on  the  hood  of  the  survey  vehicle  and  the  data 
stream  was  tied  into  the  navigation  software.  This  change  enabled  line  following  with  much  greater 
precision.  The  initial  lines  that  were  surveyed  using  the  sled  rover  were  subsequently  recollected. 

Another  incident  occurred  towards  the  end  of  the  second  day  of  surveying.  As  the  vehicle  was 
turned  after  completing  a  line,  one  of  the  fiber  reinforced  polyester  (FRP)  axles  on  the  sled  broke. 
This  issue  was  caused  because  the  wheel  collar  had  loosened,  allowing  the  wheels  to  slide  down 
the  axle  and  place  significantly  more  torque  on  it  when  turning.  The  axle  was  replaced  in  the  field 
without  any  challenges  and  it  was  subsequently  monitored  it  to  ensure  the  collar  did  not  slip  again. 
Since  then  the  axle  has  been  redesigned,  adding  an  outer  Garolite  sleeve  for  additional 
reinforcement,  and  incorporating  a  threaded  lock  nut  to  ensure  the  wheels  will  not  slide. 

During  the  second  day  of  surveying  a  GPS  problem  was  encountered  that  was  unique  to  the  site. 
One  of  the  cows  grazing  in  the  area  knocked  over  the  GPS  base  station.  Fortunately,  this  incident 
occurred  during  data  collection  and  the  driver  was  watching  the  navigation  display.  An  immediate 
shift  in  the  vehicle  position  on  the  display  let  the  driver  know  there  was  a  problem  with  the  GPS 
base.  After  halting  the  survey  and  going  to  investigate  the  reason  for  the  shift  was  discovered.  This 
problem  was  caught  quickly  enough  that  it  required  recollecting  only  a  few  lines;  however,  it  is  an 
issue  worth  noting  because  if  it  were  not  identified  by  the  driver,  it  would  not  have  discovered 
until  the  end-of-day  IVS  test.  If  not  identified  until  the  IVS,  this  problem  would  have  resulted  in 
many  more  recollects.  Because  the  navigation  display  is  the  only  real-time  indicator  of  base  station 
position,  it  should  be  monitored  during  survey  operations  to  identify  such  issues. 

A  final  issue  worth  noting  was  identified  during  the  in-field  data  quality  checks.  In  performing 
these  checks,  it  was  noticed  that  some  of  the  data  files  produced  above  normal  dynamic  noise 
profiles  in  some  of  the  late  time  channels.  Because  these  high  noise  occurrences  were  inconsistent, 
at  first  they  were  attributed  to  the  effects  of  the  pockmarked  ground  surface.  On  further  inspection, 
however,  the  problem  was  traced  to  loose  receiver  cubes.  Several  of  the  glued  inserts  to  which  the 
cubes  were  fastened  had  loosened  up  during  the  survey.  This  condition  allowed  some  of  the  cubes 
to  vibrate  slightly  (a  few  millimeters  of  movement)  when  the  sled  encountered  particularly  bumpy 
areas.  The  movement  was  enough  to  create  elevated  noise  levels  in  some  of  the  late  time  channels. 
Once  refastened,  these  affected  cubes  and  noise  levels  returned  to  the  expected  values.  While  the 
overall  classification  quality  of  the  data  was  not  impacted  significantly,  several  of  the  lines 
associated  with  the  noisier  data  files  were  recollected  (the  fasteners  have  since  been  modified  for 
these  cubes  to  ensure  they  do  not  loosen  again). 

Ultimately,  all  the  in-field  quality  checks  proved  successful.  All  data  quality  issues  were  identified 
and  the  required  data  recollected  before  demobilizing.  These  data  quality  issues  highlight  the 
importance  of  implementing  effective  data  quality  checks  as  part  of  standard  operating  procedures. 
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Because  dynamic  surveys  create  large  volumes  of  data  in  a  relatively  short  time  period  when 
compared  to  data  production  from  cued  surveys,  it  is  critical  that  appropriate  quality  checks  are  in 
place  to  ensure  that  the  field  team  departs  with  classification-quality  data. 
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